Method and apparatus for image processing

ABSTRACT

An apparatus for image processing includes a processor and a depth camera. The processor obtains a first two-dimensional image and focusing information of the first two-dimensional image, and obtains depth information of a part or all of a content captured in the first two-dimensional image via the depth camera. The processor determines a background area of the first two-dimensional image according to the focusing information. The processor also performs bokeh on the background area of the first two-dimensional image according to the depth information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/123,610, filed on Sep. 6, 2018, which claims priority to ChinesePatent Application No. 201710802117.5, filed on Sep. 7, 2017. The entirecontents of these applications are hereby incorporated by reference.

TECHNICAL FIELD

The present invention relates to a method and an apparatus for imageprocessing, and in particular, to a method and an apparatus forperforming bokeh (blurring) on the background of an image.

BACKGROUND

In a daily life, a situation usually happens that background isdisordered or not wanted when shooting a video or a static image. Thesubject may be highlighted and more vivid to generate better effect ofthe image by blurring background of the image, that is bokeh of thebackground.

In the prior art a digital camera may perform bokeh on the background bya hardware with large aperture or large zooming. The digital camerausually performs bokeh on the background by four ways as follows: 1.Make times of zooming, which are focal length, as large as possible; 2.The distance between the subject and the background is as large aspossible in shot scene. 3. The distance between the shot scene and lensis as large as possible. 4. The aperture is as large as possible whenshooting requirements are fulfilled.

In the prior art, when performing bokeh on the background therequirements for hardware are high, operation is difficult, and it isnot possible to perform bokeh quickly.

SUMMARY

The present invention provides a method and an apparatus for imageprocessing, which can solve the problem in the prior art that is whenperforming bokeh on the background the requirements for hardware arehigh, operation is difficult, and it is not possible to perform bokehquickly.

A first aspect of the present invention provides a method for imageprocessing, comprising: obtaining a first two-dimensional image andfocusing information of the first two-dimensional image, and obtainingdepth information of a part or all of content captured in the firsttwo-dimensional image, determining a background area of the firsttwo-dimensional image according to the focusing information, performingbokeh on the background area of the first two-dimensional imageaccording to the depth information.

A second aspect of the present invention provides an apparatus for imageprocessing, wherein the apparatus comprises: an obtaining module,configured to obtain a first two-dimensional image and focusinginformation of the first two-dimensional image, and obtaining depthinformation of a part or all of content in the first two-dimensionalimage; a determining module, configured to determine a background areaof the first two-dimensional image according to the focusinginformation; a background bokeh module, configured to perform bokeh onthe background area of the first two-dimensional image according to thedepth information.

A third aspect of the present invention provides a computer-readablemedium storing computer instructions that, when executed by a firstprocessor, cause the first processor to perform steps of the firstaspect of the present invention, or any of the first to twelfthimplementing way of the second aspect of the present invention.

A fourth aspect of the present invention provides an apparatus for imageprocessing, comprising a storage, a second processor and computerinstructions stored in the storage and executed by the second processor,wherein the computer instructions are executed by the second processorto perform steps of the first aspect of the present invention, or any ofthe first to twelfth implementing way of the second aspect of thepresent invention.

The method and apparatus for image processing provided by the presentinvention can reduce requirements for hardware, operate easily, andperform bokeh on the background quickly when performing bokeh on thebackground.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a flowchart of a method for image processing provided by thefirst embodiment of the present invention;

FIG. 2 is a flowchart of another method for image processing provided bythe first embodiment of the present invention;

FIG. 3 is a flowchart of another method for image processing provided bythe first embodiment of the present invention;

FIG. 4 is a structural diagram of an apparatus for image processingprovided by the second embodiment of the present invention;

FIG. 5 is a structural diagram of an apparatus for image processingprovided by the third embodiment of the present invention;

FIG. 6 is a structural diagram of an apparatus for image processingprovided by the fourth embodiment of the present invention.

DESCRIPTION OF EMBODIMENTS

The following specifically describes the technical solution in theembodiments of the present invention with reference to the accompanyingdraws in the embodiments of the present invention.

The terms “a first”, “a second”, etc. in the claims, embodiments andfigures is used to distinguish different objects and not to limitparticular order

The term “and/or” is used to describe relationship of related objects,which includes three relationships. For example, A and/or B candescribe: A only, A and B, and B only.

In the embodiments of the present invention, the word “exemplary” or“for example” is used to make an example, evidence or explanation. Theembodiments or solution described as “exemplary” or “for example” in theembodiments should not be interpreted as better or having more advantagethan other embodiments or solution. Precisely, the word “exemplary” or“for example” is used to give a conception in detailed way.

What to be explained is, for conciseness and clarity of the diagram,that the elements in the figures are not necessary to be drawn accordingto a proportion. For example, for clarity, a size of some elements canbe enlarged compared to other elements. Besides, some reference offigures can be repeated among figures in an appropriate place toillustrate corresponding or similar elements.

The following, referring to FIG. 1, is to illustrate a method for imageprocessing provided by the first embodiment of the present invention. Asshown in FIG. 1, the method includes:

Step 101, obtaining a first two-dimensional image and focusinginformation of the first two-dimensional image, and obtaining depthinformation of a part or all of content captured in the firsttwo-dimensional image.

Optionally, the depth information of a part or all of the content in thefirst two-dimensional image may include a depth map of the part or allof the content in the first two-dimensional image, or a depth value ofthe content corresponding to at least one pixel in the firsttwo-dimensional image. The depth map may be understood as a set ofpixels whose gray value represents distance between the part or all ofthe content and the picture taking device, or a set of distances betweenthe part or all of the content and the picture taking device. Thedistance between the part or all of the content and the picture takingdevice may be noted as depth value of the part or all of the content. Ifthe depth information includes a depth map of all of the content in thefirst two-dimensional image, the pixel whose gray value represents depthvalue included in the depth map and the pixel in the firsttwo-dimensional image have one-to-one corresponding relationship.

Optionally, the focusing information of the first two-dimensional imagecomprises position information of a focusing point or a focusing area ofthe first two-dimensional image. The position information of thefocusing point or the focusing area comprises coordinate information ofthe focusing point or the focusing area. The coordinate information ofthe focusing point is coordinate in the whole image in the coordinatesystem of picture taking device.

Optionally, the method further comprises obtaining a secondtwo-dimensional image of the scene in the first two-dimensional image.The obtaining depth information of the part or all of the content in thefirst two-dimensional image comprises: obtaining depth information ofthe part or all of the content in the first two-dimensional imageaccording to the first two-dimensional image and the secondtwo-dimensional image. Optionally, the obtaining depth information ofthe part or all of the content in the first two-dimensional imageaccording to the first two-dimensional image and the secondtwo-dimensional image comprises: obtaining depth information of the partor all of the content by a stereo matching algorithm according to thefirst two-dimensional image and the second two-dimensional image. Forexample, the picture taking device which shoots the firsttwo-dimensional image may be a main camera of dual cameras, and thepicture taking device which shoots the second two-dimensional image maybe a sub camera of dual cameras. Depth information of the part or all ofthe content may be obtained by a stereo matching algorithm.

Optionally, the obtaining depth information of the part or all of thecontent in the first two-dimensional image comprises: obtaining depthinformation of the part or all of the content in the firsttwo-dimensional image by a depth camera. For example, the picture takingdevice which shoots the first two-dimensional image may be an ordinarycamera, and the picture taking device which obtains depth information ofthe part or all of the content in the first two-dimensional image may bea depth camera. Optionally, the depth camera may be a depth camera in aterminal, and the depth camera may be placed side by side with thenon-depth camera which shoots the first two-dimensional image, that isthe ordinary camera. The non-depth camera may be an RGB camera.

Optionally, the method further comprises: obtaining mapping relationshipbetween depth information of the part or all of the content andcorresponding pixels in the first two-dimensional image. The“corresponding” means the pixel and the depth information arecorresponding to the same content. It is noted that, the part or all ofthe content may be an object that is represented by pixels. Optionally,the depth information comprises depth map; the obtaining the mappingrelationship between the depth information of the part or all of thecontent and the corresponding pixels in the first two-dimensional imagecomprises: obtaining the mapping relationship between coordinates ofpixels in the depth map of the part or all of the content andcoordinates of the corresponding pixels in the first two-dimensionalimage. In an actual product, the mapping relationship may be calculatedby a processor according to the obtained first two-dimensional image andthe depth information, or a mapping relationship between a depth map anda two-dimensional image may be calculated in advance and stored whichmay be used by look-up table. For example, a mapping relationshipbetween coordinate of each pixel in the depth map and coordinate of acorresponding pixel in the two-dimensional image, which is stored in aform of a table and used by look-up table.

Optionally, the obtaining the mapping relationship between thecoordinates of pixels in the depth map of the part or all of the contentand the coordinates of the corresponding pixels in the firsttwo-dimensional image comprises: obtaining mapping relationship betweenthe coordinates of the pixels in the depth map and coordinates of thecorresponding pixels of the content in a three-dimensional coordinatesystem of the depth camera; obtaining mapping relationship between thecoordinates of the pixels of the content in the three-dimensionalcoordinate system of the depth camera and coordinates of thecorresponding pixels of the content in the three-dimensional coordinatesystem of a non-depth camera which shoots the first two-dimensionalimage; and obtaining mapping relationship between the coordinates of thepixels of the content in the three-dimensional coordinate system of thenon-depth camera and the coordinates of the corresponding pixel in thefirst two-dimensional image of the content. Optionally, the methodfurther comprises: correcting the depth map according to distortioncoefficient of the depth camera. Optionally, the method furthercomprises: correcting mapping relationship between coordinates of thepixels of the content in the three-dimensional coordinate system of thenon-depth camera and coordinates of the pixels of the content in thefirst two-dimensional image according to the distortion coefficient ofthe non-depth camera.

Specifically, the mapping relationship between coordinates (u^(D),v^(D)) of a pixel in the depth map and coordinates (X^(D), Y^(D), Z^(D))of a corresponding pixel of the content in the three-dimensionalcoordinate system of the depth camera may be obtained by the followingformulas:

X ^(D)=(u ^(D) −c _(x) ^(D))/f _(x) ^(D) *Z ^(D),

Y ^(D)=(v ^(D) −C _(y) ^(D))/f _(y) ^(D) *Z ^(D);

where (u^(D), v^(D)) are the coordinates of the pixel in the coordinatesystem of the depth map, c_(x) ^(D) and c_(y) ^(D) are coordinates ofprincipal point in the x direction and y direction, respectively, f_(x)^(D) and f_(y) ^(D) are focal lengths in the x direction and ydirection, and Z^(D) is a depth value of the point which may be obtainedfrom the depth map.

Specifically, for example, the ordinary camera is an RGB camera, and themapping relationship between the coordinates of the pixel of the contentin the three-dimensional coordinate system of the depth camera andcoordinates of the pixel of the content in the three-dimensionalcoordinate system of a non-depth camera which shoots the firsttwo-dimensional image may be obtained by the following algorithm:

M ^(RGB)=[R _(relative) |T _(relative)]M ^(D);

where M^(RGB) is coordinate of the content in the coordinate system ofthe RGB camera, that is the non-depth camera, R_(relative) is a rotationmatrix between the RGB camera and the depth camera, T relative istranslation matrix between the above two cameras, and M^(D) iscoordinate of the content in the coordinate system of the depth camera.

Specifically, the mapping relationship between coordinates of a pixel ofthe content in the three-dimensional coordinate system of the non-depthcamera and coordinates of a corresponding pixel of the content in thefirst two-dimensional image may be obtained by the following algorithm:

x ^(C) =X ^(C) /Z ^(C),

y ^(C) =Y ^(C) /Z ^(C);

wherein (x^(C), y^(C)) are coordinates of the pixel of the content inthe first two-dimensional image, and (X^(C), Y^(C), Z^(C)) arecoordinates of the pixel of the content in the three-dimensionalcoordinate system of the non-depth camera.

Optionally, the obtaining the first two-dimensional image and obtainingthe depth information of the part or all of content in the firsttwo-dimensional image may include: obtaining the first two-dimensionalimage and the depth information of the part or all of content in thefirst two-dimensional image from a storing device, for example, thedepth information may be a depth map.

Step 102, determining a background area of the first two-dimensionalimage according to the focusing information.

Optionally, the determining the background area of the firsttwo-dimensional image according to the focusing information comprises:determining a main area (e.g., foreground area) of the firsttwo-dimensional image and the background area of the main area accordingto the position information of the focusing point or the focusing area.The position information of the focusing point or the focusing areacomprises coordinate information of the focusing point or the focusingarea. For example, after obtaining the coordinate information of thefocusing point, the main area may be determined by several methods, suchas background modeling, edge extraction and so on according to thecoordinate information of the focusing point, and background area may bedetermined after determining the main area. For example, it may bedetermined that an area except for the main area belongs to thebackground area, or according to the depth information it is determinedthat a set of pixels which have similar depth information belongs to thesame background area, or the background area may be determined accordingto the depth information, texture information and so on.

Optionally, the content may include at least one main area and at leastone background. For example, the main area in the first two-dimensionalimage may be a person, and the background may be a tree, a building, thesky, and land. The obtaining the depth information of the part of thecontent in the first two-dimensional image may include obtaining depthinformation of a person, and obtaining depth information of a tree and abuilding.

Step 103, performing bokeh (blurring) on the background area of thefirst two-dimensional image according to the depth information.

Optionally, the depth information of the part or all of the content inthe first two-dimensional image comprises a depth value of the contentcorresponding to at least one pixel in the first two-dimensional image;the performing bokeh on the background area of the first two-dimensionalimage according to the depth information comprises: performing differentintensity of bokeh on pixels corresponding to different depth values inthe background area of the first two-dimensional image. For example, thecontent in the first two-dimensional image may include: the main area isa person, and the background may include a tree, a car, a building andcloud. The distance between these different background and picturetaking device is from near to far. After obtaining the depthinformation, the distance between them and the picture taking device maybe determined according to the depth information, that is depth value,and the farther the background is from the picture taking device, thehigher intensity of bokeh of the background is, which means more vague.The main area of the first two-dimensional image may be not performedbokeh on. The method of bokeh may include:

Gauss filter or Box Filter (Box Filter). The depth values of pixelsincluded in the same background, such as the tree, in the firsttwo-dimensional image are not same, so different intensity of bokeh maybe determined according to different ranges of the depth values. Forexample, depth values (20-30] are corresponding to weak intensity ofbokeh, and depth values (30-40] are corresponding to strong intensity ofbokeh. Also, different background areas may be determined at first, andintensity of bokeh is determined according to an average value or themaximal value or minimum value of the depth values corresponding topixels of the background areas. Optionally, because the depth valuecorresponding to one pixel of the first two-dimensional image may beobtained, the minimum target of bokeh is the one pixel. In fact, it maybe a set of many pixels, such as a set of pixels which constitutes apart or whole of a tree.

Optionally, in fact, based on need same intensity of bokeh may beperformed on pixels which have different depth values. But as long asdifferent intensity of bokeh is performed on pixels corresponding todifferent depth values in a bokeh method, the method should fall intothe protection scope of the present invention.

Optionally, the method further comprises:

obtaining focusing indicating information, wherein the focusingindicating information instructs whether focusing is being performed;

determining that the focusing indicating information instructs focusingis being performed, obtaining a third two-dimensional image and focusinginformation of the third two-dimensional image after focusing iscompleted, obtaining depth information of a part or all of the contentin the third two-dimensional image, determining a background area of thethird two-dimensional image according to the focusing information of thethird two-dimensional image, and performing bokeh on the background areaof the third two-dimensional image according to the depth information,or,

determining that the focusing indicating information instructs thefocusing is not being performed, obtaining a fourth two-dimensionalimage which is the next frame of the first two-dimensional image,obtaining depth information of a part or all of the content in thefourth two-dimensional image, determining a background area of thefourth two-dimensional image according to the focusing information ofthe fourth two-dimensional image, and performing bokeh on the backgroundarea of the fourth two-dimensional image according to the depthinformation.

Optionally, the focusing information of the third two-dimensional imagecomprises position information of a focusing point or a focusing area ofthe third two-dimensional image, such as coordinate information of thefocusing point or the focusing area. The determining the background areaof the third two-dimensional image according to the focusing informationcomprises: determining a main area of the third two-dimensional imageand the background area of the main area according to the positioninformation of the focusing point or the focusing area. The depthinformation of the part or all of the content in the thirdtwo-dimensional image comprises a depth value of the contentcorresponding to at least one pixel in the third two-dimensional image.The performing bokeh on the background area of the third two-dimensionalimage according to the depth information comprises: performing differentintensity of bokeh on pixels corresponding to different depth values inthe background area of the third two-dimensional image. About the otherdetailed description about the third two-dimensional image in the abovemethod the aforementioned description about the first two-dimensionalimage may be referred. Similarly, About the detailed description aboutthe fourth two-dimensional image in the above method the aforementioneddescription about the first two-dimensional image may be referred. Forexample, the depth information of the part or all of the content in thefourth two-dimensional image comprises a depth value of the contentcorresponding to at least one pixel in the fourth two-dimensional image;the performing bokeh on the background area of the fourthtwo-dimensional image according to the depth information comprises:performing different intensity of bokeh on pixels corresponding todifferent depth values in the background area of the fourthtwo-dimensional image. By using the focusing indicating information, theeffect of real-time background bokeh in a video with high frame rate canbe achieved, and accuracy and stability of background bokeh may beincreased.

For example, FIG. 2 illustrates a specific process to realize thepresent invention taking dual cameras as an example. The dual camerasmay be used in a terminal, such as cell phone, notebook computer,computer, or tablet and so on. As shown in FIG. 2, after completion ofthe focusing, the first two-dimensional image obtained by a main picturetaking device and a second two-dimensional image obtained by a subpicture taking device may be obtained, and depth information may becomputed based on the first two-dimensional image and the secondtwo-dimensional image. The depth information may be computed by a stereomatching algorithm. Optionally, the depth information may be depth map.The computed depth information may be stored in a storage, which may beROM (Read Only Memory) or RAM (Random Access Memory). Then bokeh may beperformed on the background area of the first two-dimensional imageaccording to the first two-dimensional image, the obtained depthinformation and the focusing information of the first two-dimensionalimage. Then the image with background bokeh may be output. About thedetailed description of the procedure the aforementioned relativedescription may be referred to.

The followings further illustrate the aforementioned procedure to obtainthe depth information as shown in FIG. 3.

The obtaining the image shot by the main picture taking device in FIG. 3may include: obtaining the first two-dimensional image shot by the mainpicture taking device from the storage. The obtaining the image shot bythe sub picture taking device may include: obtaining the secondtwo-dimensional image shot by the sub picture taking device fromstorage. The calculating depth information may include: calculatingdepth information by a stereo matching algorithm or another algorithmbased on the first two-dimensional image and the second two-dimensionalimage, such as depth map, and outputting the depth information. Thejudging whether focusing is being performed may include: obtainingfocusing indicating information, wherein the focusing indicatinginformation instructs whether focusing is being performed, judgingwhether focusing is being performed based on the focusing indicatinginformation. When the focusing indicating information instructs focusingis being performed, waiting for completion of focusing of the mainpicture taking device and sub picture taking device, and aftercompletion of the focusing, initialization is performed. Theinitialization may include: at least one of allocating storage space,initializing parameters and other actions about shooting preparation.Then the two-dimensional images are obtained by the main picture takingdevice and sub picture taking device. When the focusing indicatinginformation instructs focusing is not being performed, a next frame ofthe first two-dimensional image and a next frame of the secondtwo-dimensional image are obtained, and the depth information iscalculated based on the next frame of the first two-dimensional imageand the next frame of the second two-dimensional image, such as a depthmap. Then the depth information is output.

By the loop shown in FIGS. 2 and 3, real-time videos with backgroundbokeh may be output.

The method disclosed by the first embodiment can reduce requirements forhardware, operate easily, and perform bokeh on the background quicklywhen performing bokeh on the background. Not only a single image can beprocessed, but effect of real-time background bokeh with high frame ratecan be achieved by using depth information with low frame rate, whichcan increase accuracy and stability of the background bokeh.

The followings illustrate an apparatus 400 for image processing providedby the second embodiment of the present invention as shown in FIG. 4,wherein the apparatus comprises:

an obtaining module 401, configured to obtain a first two-dimensionalimage and focusing information of the first two-dimensional image, andobtaining depth information of a part or all of the content in the firsttwo-dimensional image;

Optionally, the depth information of a part or all of the content in thefirst two-dimensional image may include depth map of the part or all ofthe content in the first two-dimensional image, or a depth value of thecontent corresponding to at least one pixel in the first two-dimensionalimage. The depth map may be understood as a set of pixels whose grayvalue expresses distance between the part or all of the content and thepicture taking device, or a set of distance between the part or all ofthe content and the picture taking device. The distance between the partor all of the content and the picture taking device may be noted asdepth value of the part or all of the content. If the depth informationincludes depth map of all of the content in the first two-dimensionalimage, the pixel whose gray value expresses depth value included in thedepth map and the pixel in the first two-dimensional image haveone-to-one corresponding relationship.

Optionally, the focusing information of the first two-dimensional imagecomprises position information of a focusing point or a focusing area ofthe first two-dimensional image. The position information of thefocusing point or the focusing area comprises coordinate information ofthe focusing point or the focusing area. The coordinate information ofthe focusing point is coordinate in the whole image in the coordinatesystem of picture taking device.

Optionally, the obtaining module 401, further configured to obtain asecond two-dimensional image of the content in the first two-dimensionalimage. The obtaining module 401, specifically configured to obtain depthinformation of the part or all of the content in the firsttwo-dimensional image according to the first two-dimensional image andthe second two-dimensional image. The obtaining module 401, specificallyconfigured to obtain depth information of the part or all of the contentby a stereo matching algorithm according to the first two-dimensionalimage and the second two-dimensional image. For example, the picturetaking device which shoots the first two-dimensional image may be a maincamera of dual cameras, and the picture taking device which shoots thesecond two-dimensional image may be a sub camera of dual cameras. Depthinformation of the part or all of the content may be obtained by astereo matching algorithm. Optionally, the obtaining module 401 furtherconfigured to obtain depth information of the part or all of the contentin the first two-dimensional image by a depth camera. For example, thepicture taking device which shoots the first two-dimensional image maybe an ordinary camera, and the picture taking device which obtains depthinformation of the part or all of the content in the firsttwo-dimensional image may be a depth camera. Optionally, the depthcamera may be a depth camera in a terminal, and the depth camera may beplaced side by side with the non-depth camera which shoots the firsttwo-dimensional image, that is the ordinary camera. The non-depth cameramay be an RGB camera.

Optionally, the obtaining module is further configured to obtain mappingrelationship between depth information of the part or all of the contentand corresponding pixels in the first two-dimensional image. The“corresponding” means the pixels and the depth information arecorresponding to the same content. It is noted that, the part or all ofthe content may be an object that is represented by pixels. Optionally,the depth information comprises a depth map. The obtaining module isfurther configured to obtain the obtaining module is specificallyconfigured to obtain the mapping relationship between coordinates of thepixels in the depth map of the part or all of the content andcoordinates of the corresponding pixels in the first two-dimensionalimage. In an actual product, the mapping relationship may be calculatedby a processor according to the obtained first two-dimensional image andthe depth information, or a mapping relationship between a depth map anda two-dimensional image may be calculated in advance and stored whichmay be used by look-up table. For example, a mapping relationshipbetween coordinates of each pixel in the depth map and coordinates of acorresponding pixel in the two-dimensional image, which is stored in aform of a table and used by look-up table.

Optionally, the obtaining module, specifically configured to: obtainmapping relationship between the coordinates of the pixels in the depthmap and coordinates of the corresponding pixels of the content in athree-dimensional coordinate system of the depth camera; obtain mappingrelationship between the coordinates of the pixels of the content in thethree-dimensional coordinate system of the depth camera and coordinatesof the corresponding pixels of the content in the three-dimensionalcoordinate system of a non-depth camera which captured the firsttwo-dimensional image; and obtain mapping relationship between thecoordinates of the pixels of the content in the three-dimensionalcoordinate system of the non-depth camera and the coordinates of thecorresponding pixels in the first two-dimensional image of the content.

Specifically, the mapping relationship between coordinates (u^(D),v^(D)) of a pixel in the depth map and coordinates (X^(D), Y^(D), Z^(D))of a corresponding pixel of the content in the three-dimensionalcoordinate system of the depth camera may be determined by the followingformulas:

X ^(D)=(u ^(D) −c _(x) ^(D))/f _(x) ^(D) *Z ^(D),

y ^(D)=(v ^(D) −c _(y) ^(D))/f _(y) ^(D) *Z ^(D);

where (u^(D), v^(D)) is the coordinates of the pixel in the coordinatesystem of the depth map, c_(x) ^(D) and c_(y) ^(D) are coordinates ofprincipal point in the x direction and y direction, respectively, f_(x)^(D) and f_(y) ^(D) are focal lengths in the x direction and ydirection, and Z^(D) is a depth value of the point which may be obtainedfrom the depth map.

Specifically, for example, the ordinary camera is an RGB camera, and themapping relationship between the coordinates of the pixel of the contentin the three-dimensional coordinate system of the depth camera andcoordinates of a corresponding pixel of the content in thethree-dimensional coordinate system of a non-depth camera which shootsthe first two-dimensional image may be obtained by the followingalgorithm:

M ^(RGB)=[R _(relative) |T _(relative)]M ^(D);

where M^(RGB) is coordinate of the content in the coordinate system ofthe RGB camera, that is the non-depth camera, R_(relative) is a rotationmatrix between the RGB camera and the depth camera, T relative istranslation matrix between the above two cameras, and M^(D) iscoordinate of the content in the coordinate system of the depth camera.

Specifically, the mapping relationship between coordinates of the pixelof the content in the three-dimensional coordinate system of thenon-depth camera and coordinates of a corresponding pixel of the contentin the first two-dimensional image may be obtained by the followingalgorithm:

x ^(C) =X ^(C) /Z ^(C),

y ^(C) =Y ^(C) /Z ^(C);

wherein (x^(C), y^(C)) are coordinates of the pixel corresponding to thecontent in the first two-dimensional image, and (X^(C), Y^(C), Z^(C))are coordinates of the content in the three-dimensional coordinatesystem of the non-depth camera.

Optionally, the obtaining module 401 is specifically configured toobtain the first two-dimensional image and the depth information of thepart or all of the content in the first two-dimensional image from astoring device, for example, the depth information may be a depth map.

A determining module 402, configured to determine a background area ofthe first two-dimensional image according to the focusing information.

Optionally, the focusing information of the first two-dimensional imagecomprises position information of a focusing point or a focusing area ofthe first two-dimensional image; and the determining module 402,specifically configured to determine a main area of the firsttwo-dimensional image and the background area of the area of the mainarea according to the position information of the focusing point or thefocusing area. The position information of the focusing point or thefocusing area comprises coordinate information of the focusing point orthe focusing area. For example, after obtaining the coordinateinformation of the focusing point, the area of the main area may bedetermined by several methods, such as background modeling, edgeextraction and so on according to the coordinate information of thefocusing point, and background area may be determined after determiningthe area of the main area. For example, it may be determined that anarea except for the area of the main area belongs to the backgroundarea, or according to the depth information it is determined that a setof pixels which have similar depth information belongs to the samebackground area, or the background area may be determined according tothe depth information, texture information and so on.

Optionally, the content may include at least one main area and at leastone background. For example, the main area in the first two-dimensionalimage may be a person, and the background may be a tree, a building, thesky, and land. The obtaining the depth information of the part of thecontent in the first two-dimensional image may include obtaining depthinformation of a person, and obtaining depth information of a tree and abuilding.

A background bokeh module 403, configured to perform bokeh on thebackground area of the first two-dimensional image according to thedepth information.

Optionally, the depth information of the part or all of the content inthe first two-dimensional image comprises a depth value of the contentcorresponding to at least one pixel in the first two-dimensional image;and the background bokeh module 403, specifically configured to performdifferent intensity of bokeh on pixels corresponding to different depthvalues in the background area of the first two-dimensional image. Forexample, the content in the first two-dimensional image may include: themain area is a person, and the background may include a tree, a car, abuilding and cloud. The distance between these different background andpicture taking device is from near to far. After obtaining the depthinformation, the distance between them and the picture taking device maybe determined according to the depth information, that is depth value,and the farther the background is from the picture taking device, thehigher intensity of bokeh of the background is, which means more vague.The main area of the first two-dimensional image may be not performedbokeh on. The method of bokeh may include: Gauss filter or Box Filter(Box Filter). The depth values of pixels included in the samebackground, such as the tree, in the first two-dimensional image are notsame, so different intensity of bokeh may be determined according todifferent ranges of the depth values. For example, depth values (20-30]are corresponding to weak intensity of bokeh, and depth values (30-40]are corresponding to strong intensity of bokeh. Also, differentbackground areas may be determined at first, and intensity of bokeh isdetermined according to an average value or the maximal value or minimumvalue of the depth values corresponding to pixels of the backgroundareas. Optionally, because the depth value corresponding to one pixel ofthe first two-dimensional image may be obtained, the minimum target ofbokeh is the one pixel. In fact, it may be a set of many pixels, such asa set of pixels which constitutes a part or whole of a tree.

Optionally, in fact, based on need same intensity of bokeh may beperformed on pixels which have different depth values. But as long asdifferent intensity of bokeh is performed on pixels corresponding todifferent depth values in a bokeh method, the method should fall intothe protection scope of the present invention.

Optionally, the obtaining module, further configured to obtain focusingindicating information, wherein the focusing indicating informationinstructs whether focusing is being performed; the determining module,further configured to determine that the focusing indicating informationinstructs focusing is being performed, the obtaining module, furtherconfigured to a third two-dimensional image and focusing information ofthe third two-dimensional image after focusing is completed and obtaindepth information of a part or all of the content in the thirdtwo-dimensional image, the determining module, further configured todetermine a background area of the third two-dimensional image accordingto the focusing information of the third two-dimensional image, thebackground bokeh module, further configured to perform bokeh on thebackground area of the third two-dimensional image according to thedepth information; or,

the determining module, further configured to determine the focusingindicating information instructs the focusing is not being performed,the obtaining module, further configured to obtain a fourthtwo-dimensional image which is the next frame of the firsttwo-dimensional image and obtain depth information of a part or all ofthe content in the fourth two-dimensional image, the determining module,further configured to determine a background area of the fourthtwo-dimensional image according to the focusing information of thefourth two-dimensional image, the background bokeh module, furtherconfigured to perform bokeh on the background area of the fourthtwo-dimensional image according to the depth information.

Optionally, the focusing information of the third two-dimensional imagecomprises position information of a focusing point or a focusing area ofthe third two-dimensional image, such as coordinate information of thefocusing point or the focusing area. The determining module 402, furtherconfigured to determine a main area of the first two-dimensional imageand the background area of the main area according to the positioninformation of the focusing point or the focusing area. The depthinformation of the part or all of the content in the thirdtwo-dimensional image comprises a depth value of the contentcorresponding to at least one pixel in the third two-dimensional image;and the background bokeh module 403 is specifically configured toperform different intensity of bokeh on pixels corresponding todifferent depth values in the background area of the firsttwo-dimensional image. About the other detailed description about thethird two-dimensional image in the above method the aforementioneddescription about the first two-dimensional image may be referred.Similarly, About the detailed description about the fourthtwo-dimensional image in the above method the aforementioned descriptionabout the first two-dimensional image may be referred. For example, thedepth information of the part or all of the content in the fourthtwo-dimensional image comprises a depth value of the contentcorresponding to at least one pixel in the fourth two-dimensional image.The background bokeh module 403 is specifically configured to performdifferent intensity of bokeh on pixels corresponding to different depthvalues in the background area of the fourth two-dimensional image. Byusing the focusing indicating information, the effect of real-timebackground bokeh in a video with high frame rate can be achieved, andaccuracy and stability of background bokeh may be increased.

For example, FIG. 2 illustrates a specific process to realize thepresent invention taking dual cameras as an example. The dual camerasmay be used in a terminal, such as cell phone, notebook computer,computer, or tablet and so on. As shown in FIG. 2, the obtaining module401 may be configured to obtain the first two-dimensional image obtainedby a main picture taking device and a second two-dimensional imageobtained by a sub picture taking device may be obtained, and depthinformation may be computed based on the first two-dimensional image andthe second two-dimensional image. The depth information may be computedby a stereo matching algorithm. Optionally, the depth information may bedepth map. The apparatus 400 may include storage, and the storage isused to store the computed depth information, which may be ROM (ReadOnly Memory) or RAM (Random Access Memory). Then the background bokehmodule is configured to perform bokeh on the background area of thefirst two-dimensional image according to the first two-dimensionalimage, the obtained depth information and the focusing information ofthe first two-dimensional image. Then the image with background bokehmay be output. About the detailed description of the procedure theaforementioned relative description may be referred to.

The followings further illustrate the procedure that the obtainingmodule 401 obtains the depth information as shown in FIG. 3. Theobtaining module 401 may include a two-dimensional image obtainingsubmodule, a depth information calculating submodule, and a focusingjudging submodule.

The two-dimensional image obtaining submodule is configured to obtainthe first two-dimensional image shot by the main picture taking devicefrom the storage and obtain the second two-dimensional image shot by thesub picture taking device from storage.

The depth information calculating submodule is configured to calculatedepth information by a stereo matching algorithm or another algorithmbased on the first two-dimensional image and the second two-dimensionalimage, such as depth map, and output the depth information.

The focusing judging submodule is configured to obtain focusingindicating information, wherein the focusing indicating informationinstructs whether focusing is being performed, judging whether focusingis being performed based on the focusing indicating information. Whenthe focusing indicating information instructs focusing is beingperformed, waiting for completion of focusing of the main picture takingdevice and sub picture taking device, and after completion of thefocusing, initialization is performed. The initialization may include:at least one of allocating storage space, initializing parameters andother actions about shooting preparation. Then the two-dimensionalimages are obtained by the main picture taking device and sub picturetaking device. When the focusing indicating information instructsfocusing is not being performed, a next frame of the firsttwo-dimensional image and a next frame of the second two-dimensionalimage are obtained, and the depth information is calculated based on thenext frame of the first two-dimensional image and the next frame of thesecond two-dimensional image, such as a depth map. Then the depthinformation is output.

By the loop shown in FIGS. 2 and 3, real-time videos with backgroundbokeh may be output.

The apparatus disclosed by the second embodiment can reduce requirementsfor hardware, operate easily, and perform bokeh on the backgroundquickly when performing bokeh on the background. Not only a single imagecan be processed, but effect of real-time background bokeh with highframe rate can be achieved by using depth information with low framerate, which can increase accuracy and stability of the background bokeh.

The followings illustrate an apparatus 500 for image processing providedby the third embodiment of the present invention as shown in FIG. 5. Theapparatus includes a computer-readable medium 501 storing computerinstructions that, when executed by a first processor 502, cause thefirst processor 502 to perform steps of the method in the firstembodiment.

The apparatus for image processing provided by the present invention canreduce requirements for hardware, operate easily, and perform bokeh onthe background quickly when performing bokeh on the background. Not onlya single image can be processed, but effect of real-time backgroundbokeh with high frame rate can be achieved by using depth informationwith low frame rate, which can increase accuracy and stability of thebackground bokeh.

The followings illustrate an apparatus 600 for image processing providedby the fourth embodiment of the present invention as shown in FIG. 6.The apparatus comprises a storage 601, a second processor 602 andcomputer instructions stored in the storage 601 and executed by thesecond processor 602, wherein the computer instructions are executed bythe second processor 602 to perform steps of the method in the firstembodiment. Optionally, as shown in the figure, the apparatus furtherincludes a BUS.

The apparatus for image processing provided by the present invention canreduce requirements for hardware, operate easily, and perform bokeh onthe background quickly when performing bokeh on the background. Not onlya single image can be processed, but effect of real-time backgroundbokeh with high frame rate can be achieved by using depth informationwith low frame rate, which can increase accuracy and stability of thebackground bokeh.

For example, the computer instructions may be separated to form one ormore modules/units, the one or more modules/units are stored in thestorage, and executed by the processor to complete the presentinvention. The one or more modules/units may be computer instructionssegments that can achieve a particular function, and the computerinstructions segments are used to describe a process that the computerinstructions are executed in the device/terminal.

The device/terminal may be computing device including cellphone, pad,desk computer, laptop, Personal Digital Assistant, cloud server, etc.The device/terminal may further include, but not limited to, a processoror storage. A person skilled in the art can understand the diagram ofthe present invention is only an example of the device/terminal, and nota limitation of the device/terminal. The device/terminal can includemore or less parts than what is shown in the diagram, or combination ofsome parts, or different parts. For example, the device/terminal caninclude input equipment, output equipment, network access equipment, orbus etc.

The processor may be a Central Processing Unit (CPU), another generalprocessor, a Digital Signal Processor (DSP), an Application SpecificIntegrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) oranother Programmable logical component, a discrete components gate, atransistor logical component, or a discrete hardware component. Thegeneral processor may be a micro-processor or another regular processor.The processor is the control center of the device/terminal, connectingeach part of the device/terminal by using different interfaces andcircuits.

The storage may be used to store the computer instructions and/or amodule, the processor realize various function of the device/terminal byexecuting the computer instructions and/or module stored in the storageand invoking data stored in the storage. The storage may include aninstruction sector and a data sector, where the instruction sector maystore operating system, an application program for achieving at leastone function (for example a picture viewing function) etc. Besides, thestorage may include a high speed random access memory, or a nonvolatilememory, such as hard drive, memory, plug in hard drive, Smart Media Card(SMC), Secure Digital (SD) card, Flash Card, at least one disk, Flashelement, or other Volatile solid state storage.

The modules/units integrated in the device/terminal may be stored in acomputer-readable medium when modules/units are realized in a form of asoftware function unit, and sold or used as an independent product.Based on this understanding, when realizing the present invention, apart or all of the procedure in the method of the above embodiment maybe completed by computer instructions instructing a relative hardware,where the computer instructions are stored in a computer-readablemedium, when executed by a processor, cause the processor to performsteps of any of the above method embodiments. The computer instructionsinclude computer program codes, and the computer program codes may be aform of a source code, an object code, an .exe file, or some middlestatus. The computer-readable medium may include: an entity or devicecarrying the computer program codes, a recording medium, a USB stick, amobile hard disk, a magnetic disc, a compact disc, computer storage,Read-Only Memory (ROM), Random Access Memory (RAM), electrical carriersignal, telecommunication signal, or a software distributing medium.

The image of the target object in any of the above embodiments may be animage of a part of the target object, or that of the whole targetobject. An image of a part of the target object, or that of the wholetarget object, or a variation of the image of the part of the targetobject or the whole target object is subject to the method or apparatusprovided by the present invention, where the variation does not need aninventive effort of a person skilled in the art, and falls into aprotection scope of the present invention.

What is claimed is:
 1. An apparatus for image processing, comprising aprocessor and a depth camera, wherein the processor: obtains a firsttwo-dimensional image and focusing information of the firsttwo-dimensional image, and obtains depth information of a part or all ofa content captured in the first two-dimensional image via the depthcamera; determines a background area of the first two-dimensional imageaccording to the focusing information; performs bokeh on the backgroundarea of the first two-dimensional image according to the depthinformation.
 2. The apparatus according to claim 1, wherein the focusinginformation of the first two-dimensional image comprises positioninformation of a focusing point or a focusing area of the firsttwo-dimensional image; and the processor determines a main area of thefirst two-dimensional image and the background area of the main areaaccording to the position information of the focusing point or thefocusing area.
 3. The apparatus according to claim 1, wherein theposition information of the focusing point or the focusing areacomprises coordinate information of the focusing point or the focusingarea.
 4. The apparatus according to claim 1, wherein the depthinformation of the part or all of the content in the firsttwo-dimensional image comprises a depth value of the contentcorresponding to at least one pixel in the first two-dimensional image;and the background bokeh module, specifically configured to performdifferent intensity of bokeh on pixels corresponding to different depthvalues in the background area of the first two-dimensional image.
 5. Theapparatus according to claim 1, wherein the processor obtains a secondtwo-dimensional image of the content in the first two-dimensional image;and the processor obtains the depth information of the part or all ofthe content in the first two-dimensional image according to the firsttwo-dimensional image and the second two-dimensional image.
 6. Theapparatus according to claim 5, wherein the processor obtains the depthinformation of the part or all of the content by a stereo matchingalgorithm according to the first two-dimensional image and the secondtwo-dimensional image.
 7. The apparatus according to claim 1, whereinthe processor obtains a mapping relationship between depth informationof the part or all of the content and corresponding pixels in the firsttwo-dimensional image.
 8. The apparatus according to claim 7, whereinthe depth information comprises depth map; the processor obtains amapping relationship between coordinates of pixels in the depth map ofthe part or all of the content and coordinates of the correspondingpixels in the first two-dimensional image.
 9. The apparatus according toclaim 8, wherein the processor: obtains a mapping relationship betweenthe coordinates of the pixels in the depth map and coordinates ofcorresponding pixels of the content in a three-dimensional coordinatesystem of a depth camera; obtains a mapping relationship between thecoordinates of the pixels of the content in the three-dimensionalcoordinate system of the depth camera and coordinates of correspondingpixels of the content in the three-dimensional coordinate system of anon-depth camera which captured the first two-dimensional image; andobtains a mapping relationship between the coordinates of the pixels ofthe content in the three-dimensional coordinate system of the non-depthcamera and the coordinate of the corresponding pixel in the firsttwo-dimensional image of the content.
 10. The apparatus according toclaim 1, wherein: the processor obtains focusing indicating information,wherein the focusing indicating information indicates whether focusingis being performed; the processor determines that the focusingindicating information indicates focusing is being performed; theprocessor obtains a third two-dimensional image and focusing informationof the third two-dimensional image after focusing is completed andobtain depth information of a part or all of the content in the thirdtwo-dimensional image; the processor determines a background area of thethird two-dimensional image according to the focusing information; andthe processor performs bokeh on the background area of the thirdtwo-dimensional image according to the depth information of the part orall of the content in the third two-dimensional image, or wherein: theprocessor determines the focusing indicating information indicates thefocusing is not being performed, the processor obtains a fourthtwo-dimensional image which is the next frame of the firsttwo-dimensional image and obtain depth information of a part or all ofthe content in the fourth two-dimensional image, the processordetermines a background area of the fourth two-dimensional imageaccording to the focusing information of the fourth two-dimensionalimage, and the processor performs bokeh on the background area of thefourth two-dimensional image according to the depth information.
 11. Theapparatus according to claim 10, wherein the depth information of thepart or all of the content in the third two-dimensional image comprisesa depth value of the content corresponding to at least one pixel in thethird two-dimensional image; and the processor performs differentintensity of bokeh on pixels corresponding to different depth values inthe background area of the third two-dimensional image.
 12. Theapparatus according to claim 10, wherein the depth information of thepart or all of the content in the fourth two-dimensional image comprisesa depth value of the content corresponding to at least one pixel in thefourth two-dimensional image; and the processor performs differentintensity of bokeh on pixels corresponding to different depth values inthe background area of the fourth two-dimensional image.